Computing Inter-Rater Reliability With the SAS System

The SAS system V.8 implements the computation of unweighted and weighted kappa statistics as an option in the FREQ procedure. A major limitation of this implementation is that the kappa statistic can only be evaluated when the number of raters is limited to 2. Extensions to the case of multiple raters due to Fleiss (1971) have not been implemented in the SAS system. A SAS macro called MAGREE.SAS, that can handle the case of multiple raters is available at the SAS Institute’s web site (check it at http://ftp.sas.com/techsup/download/stat/magree.sas). In this article, we discuss about the use of the SAS system to compute Kappa statistics in general. We will also present our SAS macro called INTER RATER.MAC that can handle multiple raters and can compute AC1 and Kappa statistics (overall and for each category) as well as the associated standard errors. In the INTER RATER.MAC SAS macro, the AC1 standard error is calculated both conditionally on the sample od raters as well as unconditionally. The unconditional standard error has the particular feature of taking into account the additional variability that is due to the sampling of raters.